Device and method for combined display of angiograms and current x-ray images

ABSTRACT

The invention relates to a device and to a method for superimposed display of current (X-ray) image (A) of an object ( 8 ), such as a catheter for example, and a map image (B) of the vascular system. In this connection, for map images (B) archived in a memory ( 6 ) the associated distance images (D) are calculated by means of a distance transformation (A). In the current image (A) the object ( 8 ) is segmented (Σ). By means of the distance image (D), a transformation of the map image (B) is then calculated, so that, when the current image (A) and the transformed map image (Θ(B)) are superimposed on a monitor ( 10 ), the image of the object ( 8 ) lies in the path network of the transformed map image.

The invention relates to a device and to a method for combined displayof a current image of an object, which is located in a path network suchas in particular the vascular system of a patient, and a map image ofthe path network.

The combination of a current image of an object and a map image of theobject surroundings is performed, for example, when navigating acatheter through the vascular system of a patient. The underlyingproblem will therefore be explained subsequently with the aid of anexample of a cardiac catheter examination, although the presentinvention is not restricted to this area of application. In the case ofthe systems customarily used for cardiac treatment, static angiogramsand fluoroscopic images currently being recorded are displayed on twodifferent monitors side by side. Angiograms are here depictions of thevascular system, in which the vessels are displayed highlighted, forexample by administering a contrast medium. In the case of thesesystems, it is left to the doctor carrying out treatment to relate theposition of an object, such as, for example, a catheter or a guide wire,recognizable on the current picture, to the map image of the vascularsystem, that is, to superimpose in his mind the two monitor imagesdisplayed side by side.

In this context, a device is known from JP-A-2002-237996, in which acurrent fluoroscopic image and a static vascular map are superimposed onthe same monitor. The difficulty with such superimpositions is thatowing to an overall movement of the patient, as well as his heartbeatand breathing, the position and form of organs in the current imagesconstantly change, so that to some extent considerable geometrical andanatomical discrepancies exist between the superimposed images. Toalleviate this problem, data banks containing static vascular maps fromdifferent phases of the cardiac and/or respiratory cycle can be used inorder by means of an electrocardiogram (ECG) and/or the measuredrespiration phase to allocate to a current fluoroscopic image the staticvascular map (from the same or a similar cardiac or respiratory cycle)that is the best match for it. Even when using such advanced methods,geometric discrepancies between the superimposed images still remain,which can seriously impair the optical impression and consequently theusefulness of the superimposition. Furthermore, for parts of the bodythat are not subject to cyclical spontaneous movement (for example, thehead, the extremities), the quality of the superimposition is also poorif map recording and imaging of the intervention procedure are carriedout at separate times, because patient movements between recordings are,in the main, inevitable, and even reproduction of the image geometry islimited mechanically.

Improved superimpositions of current recordings and map images could inprinciple be achieved by transformations, which bring common imagecontents into register. Such methods, known as multimodalityregistration in the literature, can nevertheless not be applied in theabove cases as a rule, since the map image of the path network and thecurrent recording of an object in the path network have no relevantcommon image content. In particular, the objects in the path network(e.g. catheter, guide wire) correspond neither in form nor in appearancewith the path network itself (contrast medium-filled blood vessels).

Against this background, it was an object of the present invention toprovide means for improved, real-time combined display of a currentimage of an object and a map image of the path network in which theobject is located.

That object is achieved by a device having the features of claim 1 aswell as by a method having the features of claim 11. Advantageousembodiments are contained in the subsidiary claims.

The device according to the invention is used for combined display of animage of an object, which is located in a path network, and a map imageof the (especially form-changing) path network. The first-mentionedimage is referred to hereinafter as the “current image”, without arestriction being associated with this in relation to specific timeperiods. Neither are there any fundamental restrictions in respect ofthe dimensionality of the current image and the map image (1D, 2D, 3D,4D, . . . ). The object can be, for example, a catheter or anintervention device (guide wire, stent, balloon) on a catheter, and thepath network can be correspondingly the vascular system of a patient.Alternatively, however, the object can be, for example, a capsulelocated in the gastrointestinal tract of a patient, or a non-medicalapplication can be involved. The typical feature is that the object isable to move only along the paths allowed by the path network. The mapimage preferably displays the path network in highlighted form. Forexample, the map image can be an angiogram that has been prepared fromthe vascular system of a patient to whom contrast medium has beenadministered. The device contains a data-processing system, which isarranged to perform the following steps:

In a map image to identify the path network by suitable segmentation.Segmentation is understood here in conventional manner to mean theassignment of pixels to different classes or objects. In the presentcase, the segmentation is able to determine in particular for each pixelof the map image whether it belongs to the path network or not.Segmentation can be effected fully automatically or alternatively wherenecessary semi-automatically, in other words, by interactive userintervention.

From the above-mentioned segmentation result to calculate auxiliaryinformation and to archive it in the memory of the data-processingsystem, from which auxiliary information a transformation that bringsthe object and path network into register can be determined in real timefor every possible position of an object in the image. What positions ofthe object are “possible” will depend primarily on the underlyingapplication; in the extreme case, all possible positions on the imagearea can be regarded as eligible. Subject to the method used in step d),the information needed to be able to discover as quickly as possible thenearest plausible location in the path network for the possiblepositions of an object in the image is determined in advance in theauxiliary information.

The auxiliary information can in particular be in the form of an(auxiliary) image of the region of the path network. For a givenposition of an object, at the corresponding point of the auxiliary imageit is then possible directly to remove information that is needed todetermine a transformation in real time.

To segment from the current image a relevant object that is located inthe path network. The fact that the object is located in the pathnetwork emerges typically not from the current image, but is based onthe general conditions of the underlying application.

Using the auxiliary information from step b), to determinetransformations of the map image and the current image, so that when thetransformed map image and the transformed current image aresuperimposed, the image of the object comes to lie in the path networkof the transformed map image. One of the mentioned transformations, e.g.that of the current image, is typically defined by the identity, so thatonly the map image is subjected to a “real” transformation. Thetransformations can incidentally be of any kind, that is, in particularlinear or non-linear. In particular, a translation, a rotation and/or ascaling can be involved.

With the device described, it is possible to achieve an adjustment,based on an object such as a catheter for instance, of thesuperimposition of a current image and a map image, the constraint beingexploited that the observed object must be located at all times in thepath network. The superimposed images are therefore transformed in sucha way that the path network displayed on the map image lies over theobject displayed on the current image. In this way, registration on thebasis of image contents (vessel, catheter etc.) is achieved, and themismatches, particularly irritating for the user, when an object ofinterest does not lie or does not lie exactly in the path network, canbe avoided. Moreover, it is important for the device that respectiveauxiliary information is calculated in advance for the map image used,the auxiliary information containing information about the path networkand extending this information, for example, over the entire image area,so that it is immediately retrievable during the later intervention.This ultimately enables the superimposition to be carried out in realtime, which is an indispensable prerequisite for maximum clinicalusefulness of the device.

As was already mentioned, the auxiliary information can comprise inparticular one or more images of the region of the path network. In thisregard, the auxiliary information comprises preferably a distance imagein relation to the path network, which is obtained from the particularmap image by a distance transformation. A distance transformation is anoperation known from digital image-processing (cf. Jähne, DigitaleBildverarbeitung, 5^(th) edition, Chapter 18, Springer Verlag BerlinHeidelberg, 2002). Here, a pixel of the distance image can in particularcontain information about in which direction and/or at what distancefrom that point a specific segmentation object exists. Such a distanceimage is especially well suited for rapid determination of the requiredtransformations, since it contains implicitly for each pixel themagnitude of the necessary displacement into the path network. In theimportant cases of application, in which the map image is known inadvance, the associated distance image can be calculated in advance andstored in a memory. Later on, this enables calculation of thetransformations to be carried out in real time during an ongoingintervention.

According to a preferred embodiment of the device, the data-processingsystem is arranged to perform the following individual steps:

b1) Determination of the position of the image of the object in thecurrent image. For example, by segmentation, the position of a catheteror rather its tip can be determined in a fluoroscopic X-ray image. Apartfrom an individual point, the segmentation result can also contain anentire object, which in the superimposed view is supposed to lie as faras possible in the path network (a complete match is not always possiblein the case of rapid, rigid transformations, especially in the case ofbiological path networks);

c1) Determination of the shortest displacement that in the best possiblemanner will transfer into the path network the position in the distanceimage that corresponds to the above-mentioned position of the objectimage in the current image. In other words, first of all one determinesthe corresponding position of the object image in the distance image isproduced, which occurs when the position of the object image in thecurrent image is transferred “one to one” or conforming to the geometricrelations between the current image and the map image known fromrecording parameters. Normally, this corresponding position will liecompletely or partially outside the path network, since a path networksuch as e.g. the vascular system is subject to constant displacement anddeformation and therefore does not normally exist at the same point andin the same configuration on the map image and the current image;

c2) Identification of a transformation of the map image and/or of thecurrent image that includes the above-mentioned displacement. Thistransformation can extend the displacement in particular globally to anentire image. The displacement can alternatively, however, be continuedlinearly or non-linearly, such that specific marginal conditions, forexample, the invariance of the image edges, are satisfied.

In a preferred version of the device, the data-processing system isarranged to carry out a segmentation of the path network in the mapimage and at the same time to assign to each pixel of the map image aprobability that it belongs to the network. In other words, aprobability-based segmentation is carried out, in which the pixels arenot sorted strictly into just one of two classes (belonging to theobject or not), rather, only probabilities for an affiliation areassigned. This procedure better suits in particular the situation whenprocessing medical data, since there, on account of the complexity ofthe structures depicted and the restricted image quality, generallyspeaking no really reliable decision can be made about the affiliationto a vessel or the like. At the same time, a meaningful gauge of thereliability of a result obtained can also be defined by theprobability-based segmentation.

The device can in particular contain an imaging arrangement, for examplean X-ray apparatus and/or an MRI apparatus, with which the current imageof the object can be produced. Furthermore, the imaging arrangement canserve to generate also the map images of the residence region of theobject. Such a device is especially suitable for navigation of acatheter during medical examinations. The device can also contain morethan one imaging device, for example, an X-ray apparatus and an MRIapparatus, so that the current recording and the map image(s) canoriginate from different modalities.

According to a further aspect of the device, this contains a memory forstoring a number of map images, the map images being categorizedaccording to a varying state of the path network. In this instance it ispossible to select from among the several map images an optimum mapimage for the combination to be effected.

The device contains furthermore preferably a sensor device for detectingat least one parameter that describes a varying state of the pathnetwork of the object. In particular, the sensor device can be arrangedto detect an electrocardiogram and/or the respiratory cycle of a patientundergoing examination. Such a sensor device can be used in conjunctionwith the above-mentioned memory for a number of map images, in order onthe one hand to categorize the stored map images according to theassociated state of the path network and in order on the other hand todetermine the state of the path network pertaining to the current image.

In conjunction with the above-mentioned embodiment of the devicecontaining a memory, the data-processing system can furthermore bearranged to select from the memory of the device that map image of whichthe “index” or associated state of the path network is the best possiblematch for the state of the path network that existed as the currentimage was being taken. If, for example, the memory contains several mapimages of the vascular system of a patient at different phases of thecardiac cycle, one can select from these the one that comes from thesame phase of the cardiac cycle as the current image. In this manner itis possible to take into account parameterizable and especially cyclicalspontaneous movements of the path network and from the outset to combinethe current image only with a map image that is the best possible match.

The device can in particular contain a display device linked to thedata-processing system, on which the transformed map image is displayedsuperimposed entirely or in sections on the transformed current image ora section thereof. In the context of a catheter investigation, a doctor,for example, can then observe on the monitor fluoroscopic live images ofthe catheter, which at the same time show him the vascular structurearound the catheter as a section of a vascular map.

The invention relates furthermore to a method for combined display of acurrent image of an object, which is located in a path network, and amap image of the path network, comprising the following steps:

a) segmentation of the path network in the map image;

b) calculation and storage of auxiliary information from thesegmentation result, wherein for every possible position of an object inthe image a transformation that brings the object and path network intoregister can be determined in real time from the auxiliary information;

c) segmentation of a relevant object that is located in the path networkfrom the current image;

d) determination of transformations of the map image and the currentimage using the auxiliary information, so that, when the transformed mapimage and the transformed current image are superimposed, the image ofthe object comes to lie in the path network of the transformed mapimage.

The method implements in a general form the steps that can be performedwith a device of the kind described above. For an explanation of thedetails, advantages and further aspects of the method, the reader istherefore referred to the above description. These and other aspects ofthe invention are apparent from and will be elucidated, by way ofnon-limitative example, with reference to the embodiments describedhereinafter.

In the drawings:

FIG. 1 shows the components of a device according to the invention forsuperimposed display of two images;

FIG. 2 is an illustration of an example distance image.

In the case of the medical application illustrated in the Figure as arepresentative example, the movement of a catheter 2 or more preciselyof the catheter tip and/or a guide wire 8 in the vascular system 9 of apatient 1 is to be observed. For that purpose, fluoroscopic X-raysimages of the body volume being examined are produced with an X-rayapparatus 4, and are transferred as current images A to adata-processing system 5. The difficulty with such fluoroscopic imagesis that the vascular system 9 does not usually stand out thereon, sothat with this system reliable navigation of the catheter or a guidewire to a specific location within the vascular system is hardlypossible. A better display of the vascular system could, admittedly, beachieved by injection of a contrast medium, but such measures must beused as sparingly as possible, owing to the stress associated therewithfor the patient.

To improve catheter navigation, in the case of the system illustratedseveral angiograms B are prepared with the X-ray apparatus 4 before orduring the actual catheter examination and are stored in a memory 6 ofthe data-processing system 5. The angiograms can be produced, forexample, by injections of contrast medium, so that the vascular tree ofthe patient can easily be seen on them. They are therefore hereinafterreferred to also as “map images or “vascular maps” (road maps).

Since the heartbeat has significant effects on the position and form ofthe vascular system of the heart and the adjoining organs, map images Bfrom different phases of the cardiac cycle of the patient 1 are archivedin the memory. The cardiac phase belonging to a particular map image Bis here indicated by an electrocardiogram, which is recorded by anelectrocardiograph 3 in parallel with the X-ray images. Furthermore, mapimages can be prepared also at different phases of the respiratorycycle, which is detected by a respiration sensor such as a chest belt orsimilar. For the sake of clarity, such an additional or alternativeindication of the map images B by way of the respiratory cycle is notspecifically shown in the Figure. The map images B could be subjected tofurther techniques for image improvement in order to improve the imagequality for the superimposition.

During the catheter examination carried out for therapeutic ordiagnostic purposes, fluoroscopic images A of the catheter tip or aguide wire 8 are continuously produced and passed together with theassociated ECG to the data-processing system 5. The phase of theelectrocardiogram or of the cardiac cycle pertaining to a current imageA is then established by the data-processing system 5, and the map imageB that matches this cardiac phase best is selected from the memory 6.

The current image A and the map image B can in principle be displayedside by side on two different monitors or superimposed on one another onthe same monitor. Since the map image B to the matching cardiac phasewas selected, the geometrical or anatomical correspondence between theimages A, B thus superimposed would already be a comparatively good one.Nevertheless, because of parallax in the image production, because ofsoft tissue movement and as a result of similar influences, in practiceslight discrepancies always appear between the superimposed aggregateimages, and cannot be eliminated by transformations without analysis ofthe current image content. These discrepancies can be visually verydisruptive and considerably reduce the usefulness of thesuperimposition.

To improve the image quality during the superimposition of two images, aregistration method based on the position on the object to be imaged,that is to say, primarily the catheter or guide wire 8, is proposed.Within the scope of this method, in the map images B the vascular treeis roughly pre-segmented. Segmentation in image processing is understoodto mean the assignment of pixels to objects.

In this connection, the registration method requires the selection of asuitable method for segmentation and a suitable method for preparationof the segmentation result, in order to aid a subsequent fastregistration with objects in the vascular system. Both choices are to beeffected with regard to a quick and robust algorithm for discovering thebest-possible match between path network and current object. For thesegmentation of blood vessels, the principle axis transformation of thelocal Hessian matrix (Schrijver M; “Angiographic image analysis toassess the severity of coronary stenoses”, Twente university press,Enschede, 2002) is suitable. Since in the case of real X-ray images ofthe vascular system it is not normally possible to assign a pixelreliably to a vessel, a probability-based segmentation is preferablyeffected here. In this, each pixel is assigned a value that describesthe probability that the pixel belongs to a vessel. A multiplicativedistance transformation with a hyperbolic mask, in which entriesdecrease with the inverse of the distance to the center, allows simplegradient descent optimizations even for complex path networks such avascular trees having pathological modifications. Such a distance imageD indicates locally in what direction from or at what distance from thepoint under consideration there is a greater probability of the presenceof a vessel. The distance image D can be displayed visually by a heightrelief across an image area, the height of the points of the reliefrepresenting the distance to the vascular system. FIG. 2 shows in thisconnection the two-dimensional projection of the contours of an examplerelief. Calculation of the probability-based map images B and theassociated distance images D can advantageously be effected off-line orin advance, the results being held in the memory 6. During a real-timeapplication, such as the medical examination under consideration forexample, these calculations do not impede implementation of the method.

After selecting from the memory 6 the map image B that best matches thecurrent image A, the distance image D pertaining to this map image B isused to estimate the position of the object 8 of interest (catheter orguide wire) on the map image B. For that purpose, first of all, the(radio-opaque) object 8 is segmented in the current image A using asuitable segmentation method Σ. There are various algorithms availablehere, from which an optimum variant can be selected with respect to theunderlying application, the intervention device being displayed as wellas the real-time efficiency (Baert S A M, Niessen W J, Meijering E H W,Frangi A F, Viergever M A: “Guide wire tracking during endovascularinterventions”, Proc. 3^(rd) MICCAI, 2000).

By a simple and quick gradient descent the distance image D can then bedisplaced so that the overlap between the position of the object 8 andthe vessel regions becomes maximum. At the same time, only rigiddisplacements (shifts and/or rotations) of the segmented object relativeto the map image B can be permitted, although non-linear transformationscan be included as well if this has advantages in the specificapplication. The resulting transformation Θ is then applied to the mapimage B, and the transformed map image Θ(B) is then displayed on themonitor 10 superimposed on the current image A. In the resultingcombined image C, the intervention device 8 is clearly visible to thedoctor in a high-contrast vascular tree, whereby navigation of theinstrument and placement of surgical treatment is appreciablyfacilitated.

Moreover, in the case of the combined display on the monitor 10, justone section of the map image B and/or one section of the current image Ain the region of the object 8 can be used, in order, by limiting theregistration region, to improve accuracy compared with a globalregistration.

1. A device for combined display of a current image (A) of an object(8), which is located in a path network (9), and a map image (B) of thepath network (9), the device containing a data-processing system (5)that is arranged a) in a map image (B) to identify the path network bysegmentation; b) to calculate from the segmentation result auxiliaryinformation (D) and archive it in the memory of the data-processingsystem, from which a transformation (Q) that brings the object and pathnetwork into register can be determined in real time for every possibleposition of an object in the image; c) from the current image (A) tosegment a relevant object (8) that is located in the path network (9);d) using the auxiliary information (D), to determine transformations (Q)of the map image (B) and of the current image (A), so that, when thetransformed map image (Q(B)) is superimposed on the transformed currentimage (A), the image of the object (8) comes to lie in the path networkof the transformed map image.
 2. A device as claimed in claim 1,characterized in that the auxiliary information includes a distanceimage (D) in relation to the path network (9), which is obtained fromthe particular map image (B) by a distance transformation (D).
 3. Adevice as claimed in claim 2, characterized in that the data-processingsystem (5) is arranged b1) to determine the position of the image of theobject (8) in the current image (A); c1) for the position correspondingthereto in the distance image (D), to determine the shortestdisplacement leading into the path network (9); c2) to identify atransformation (Θ) of the map image (B) and/or of the current image (A)that includes the determined displacement.
 4. A device as claimed inclaim 1, characterized in that the determined transformations (Θ)include a tanslation, a rotation and/or a scaling.
 5. A device asclaimed in claim 1, characterized in that the data-processing system (5)is arranged during segmentation of the path network (9) in the map image(B) to assign to each pixel a probability that it belongs to the network(9).
 6. A device as claimed in claim 1, characterized in that itcomprises an imaging arrangement, especially an X-ray apparatus (4)and/or an MRI apparatus, for recording the current image (A) andoptionally the map image (B).
 7. A device as claimed in claim 1,characterized in that it comprises a memory (6) for storing a number ofmap images (B), which are categorized according to a varying state ofthe path network (9).
 8. A device as claimed in claim 1, characterizedin that it comprises a sensor device (3) for detecting at least oneparameter that describes a varying state of the path network (9),preferably for detecting an electrocardiogram and/or the respiratorycycle.
 9. A device as claimed in claim 6, characterized in that thedata-processing system (5) is arranged to select from the memory (6) amap image (B) of which the associated state of the path network (9) isthe best possible match for the state of the path network (9) during thecurrent recording (A).
 10. A device as claimed in claim 1, characterizedin that it contains a display device (10) and the data-processing system(5) is arranged to display on the display device (10) the transformedmap image (Θ(B)) superimposed entirely or in sections on the transformedcurrent image or a section thereof.
 11. A method for combined display ofa current image (A) of an object, which is located in a path network(9), and a map image (B) of the path network (9), comprising thefollowing steps: a) segmentation of the path network in the map image;b) calculation and storage of auxiliary information from thesegmentation result, wherein for every possible position of an object inthe image a transformation that brings the object and path network intoregister can be determined in real time from the auxiliary information;c) segmentation of a relevant object that is located in the path networkfrom the current image; d) determination of transformations of the mapimage and of the current image using the auxiliary information, so that,when the transformed map image and the transformed current image aresuperimposed, the image of the object comes to lie in the path networkof the transformed map image.